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Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 
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1)13 Responsive to communication(s) filed on 24 June 2005 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
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4) D Claim(s) is/are pending in the application. 
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5) Q Claim(s) is/are allowed. 

6) E! Claim(s) 1^27 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
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application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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Response to Arguments 
Applicant's arguments filed 6/24/05 have been fully considered but they are not 
persuasive. 

Applicant on page 2 of the remarks line 15 argues that the Examiner's question does not 
fit under rejection 1 12, second paragraph as indefinite, e.g. "does Applicant's computerized 
method analyze the content of the multimedia prior to storage area or after the storage area?" 

Examiner's comment: Examiner believes that the answer to the question is unclear/vague 
and simply asking Applicant to clarify the nature of the unclear subject, see next paragraph. 
Examiner's comments on other questions: The method of graph rewriting contains rules and 
models (i.e. pre-defined template graphs), which are located in fig. 1, GCS 103. Examiner found 
the closest answer to the question from the specification as follows: the question is [What does 
occur to "validating the input graph" if there is no match with a template graph?] the answer is 
the invalid graph goes to "EXIT" see fig. 2. The next two questions [Is the graph rewriting 
exactly similar to the multimedia content?] the Examiner assumes the answer is "NO", because 
Applicant in fig. 1 at GCS 103 has his own rules and models. The next question is [Does the 
comparing scenario identify all objects for multimedia content?] The Examiner assumes the 
answer is "YES" because according to fig. 2 all objects are contained edges and nodes. 
In respect to Examiner's comments associated with the questions above, the rejection of claims 
1-27 under 35 U.S.C. 1 12, second paragraph has been withdrawn. 

Applicant on pages 2-3 argues that the Examiner has not considered Applicant's claims in their 
entirety. Applicant emphasizes that the invention is directed toward graph rewriting. Applicant 
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provides the definition for the Graph rewriting i.e. the process of combining graphs, replacing 
nodes with graphs, replacing edges with more complex configurations, and generating series of 
"production steps" to move from one graph to another. Applicant on page 3, line 9 argues the 
reference Petrakis does not disclose and rearranging of an ARG after it is originally created from 
the medical image. 

Examiner's reply: As well known in the art of Computer Science, the broad term "multimedia" in 
claim 1, line 4 can combine text, graphics, full-motion video, and sound into an integrated 
package. Applicant on the same page adds that the medical images are well known as being only 
still pictures and thus cannot be properly interpreted as multimedia data. Petrakis on page 438 
under section 3. 1 teaches that image content can be described indirectly through attributes (e.g., 
subject, speaker, etc.) or text (e.g., captions). Queries by image content require that, prior to 
storage, images are processed, and appropriate descriptions of their content are extracted and 
stored in the database. A person skill in the art would have been interpreted the image retrieval 
by content e.g. captions, speaker i.e. audio, subject i.e. video and etc. Therefore all the medical 
images are not only still pictures and thus can be properly interpreted as multimedia data. 
Examiner's suggestion: Applicant may specify the type of multimedia content in claim 1. 
According to the Applicant's definition of the Graph rewriting [i.e. the process of combining 
graphs, replacing nodes with graphs, replacing edges with more complex configurations, and 
generating series of "production steps" to move from one graph to another]. Examiner refers 
Applicant to see e.g. fig. 1 on page 436 that illustrates a sketch of a face and its corresponding 
ARG. Examiner compares fig. 1 of the present invention (hereinafter referred as 01) with fig. 1 
of the reference (hereinafter referred as 97), as follows: in 01 step 107 i.e. input graph, can be 
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interpreted as 97 (a) graph of a face i.e. input graph. In 01 step 101 is considered as GRS with 
rules and models e.g. predefine template graphs. Examiner interpretation: A rewrite system (e.g. 
Applicant refers it as GRS and the reference referred it as ARG) that is a set of equations over a 
given set of terms called rules that characterize a system of computation. Now back to the 
figures, in 01 step 103 the rules and models i.e. equivalent to 97 part (b), or at page 437 
equations 1-4 defined some of the rules and the model under section 3. 1 of the reference. The 
step of 109 at 01 is equivalent to 97, part (b). 

Examiner's suggestion: Applicant claims a computerized method of graph rewriting but uses 
broad claim languages that can be easily anticipated by the application of the graph rewriting i.e. 
the work of Petrakis in the medical area. Applicant may overcome the reference by limiting the 
claim languages e.g. input graph (TV signal, Video signal etc.); multimedia content (Text, 
Audio, etc.) ; pre-defined template and method of validating (equation, geometry etc.). 
For the reasons above the previous rejection is still maintained. 
Examiner encourages Applicant to schedule an interview. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

Claims 1-27 rejected under 35 U.S.C. 102(b) as being anticipated by Euripides G.M. 
Petrakis, Member, IEEE, and Christos Faloutsos (hereafter refers as a "Euripides"). 
Claim 1. 
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Euripides discloses on page 435 under section 1, that shows a computerized method, quote 
from the reference: In medicine, in particular, a large number of images of various imaging 
modalities (e.g., computer tomography, magnetic resonance, etc.) are produced daily and used 
to support clinical decision making. A computerized method of graph rewriting, a graph having 
nodes representing entities and edges between the nodes representing relationships between 
entities, the method comprising: Euripides in figs. 1-2 illustrates graph rewriting of an image 
and its corresponding ARG (Le. graph rewriting). Euripides on page 436 under section 2.1 
discloses image descriptions are given in terms of object properties, and in terms of 
relationships between objects. The textbook approach to capture this information is the 
Attributed Relational Graphs (ARGs). In an ARG, graph nodes represent the objects, and arcs 
between such nodes represent the relationships between objects. Both nodes and arcs are 
labeled by attributes corresponding to properties (features) of objects and relationships, 
respectively. 

As claim invention discloses that comparing an input graph representing a description scheme 
for multimedia content with a set of predefined template graphs. Euripides on page 437 in first 
column and in fig. 2 discloses the specific features, which are used in ARGs, are derived from 
the raw image data and, depending on the application, can be geometric (Le., independent of 
pixel values), statistical, or textural, or features specified in some transform domain (e.g., 
Fourier coefficients of object shapes). In the case of medical CT and MRI images used in this 
work, the set of features is given in Section 4.1. However, the proposed methodology is 
independent of any specific kind of features. The problem of retrieving images, which are 
similar to a given example, image is transformed into a problem of searching a database of 
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stored ARGs: Given a query, its ARG has to be computed and compared with all stored ARGs. 
Matching between ARGs is a well-known problem and has been studied extensively in the 
computer vision literature. Specifically, matching a query and a stored graph is treated as a 
sub graph isomorphism problem. 

As claim invention discloses in the last line of claim one that validating the input graph when 
there is a match with a template graph. Euripides on page 437 in second column discloses that 
the first term in Eq. (1) is the cost of matching (validating as the claim language specifies) 
associated nodes, while the second term is the cost of matching the relationships between such 
nodes. In our setting, only a subset of the objects in the stored image S needs to be matched 
There is no cost if the data image contains extra objects; however, we assume that the cost is 
infinite if the data image is missing one of the objects of the query. COST is the cost of 
matching features of objects or features of relationships between associated objects. The 
distance between images Q and S is defined as the minimum distance computed over all 
possible mappings FQ: see Eqs. 2-4. Similarity searching in an IDE of stored ARGs requires 
that all images within distance t must be retrieved. Specifically, we have to retrieve all the 
images S that satisfy the Eq. 4 condition. 
Claim 2. 

The computerized method of claim 1, wherein the comparing uses a graph matching 
process. Euripides on page 436 under section 2.1 discloses image descriptions are given in 
terms of object properties, and in terms of relationships between objects. The textbook 
approach to capture this information is the Attributed Relational Graphs (ARGs). In an ARG, 
graph nodes represent the objects, and arcs between such nodes represent the relationships 
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between objects. Both nodes and arcs are labeled by attributes corresponding to properties 
(features) of objects and relationships, respectively. 
Claim 3. 

The computerized method of claim 2, wherein the comparing comprises: creating 
adjacency matrices representing the input graph and the set of template graphs. 
Euripides on page 440 under section 4.1 and in second column teaches that additional 
features that could be used include the average gray-level and texture values, moments, or 
Fourier coefficients, etc, as object descriptors; relative size, amount of overlap-ping, or 
adjacency, etc, can be also used to characterize the relationships between objects. 

Claim 4. 

The computerized method of claim 1 further comprising: evaluating the input graph against a set 
of pre-defined alphabet graphs; and applying a rule associated with a matching alphabet graph to 
the input graph, the rule represented by a rule graph and a set of morphism graphs. Euripides on 
page 437 in column one discloses matching a query and a stored graph is treated as a sub 
graph isomorphism problem. Examiner comment: the meaning of morphism is an abstraction 
of a function or mapping between two objects. 
Claim 5. 

The computerized method of claim 4, wherein the evaluating uses a graph matching 
process. This step is inherent in the reference see figs. 1-3 of the Euripides. 

Claim 6. 

The computerized method of claim 5, wherein the evaluating comprises: creating 
adjacency matrices for the input graph and the set of alphabet graphs. Euripides on page 439 in 
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the first column discloses that the CAFIIR system proposes the "iconic index tree" to 
accelerate the search on facial images. One novelty of the system is that it can process "fuzzy" 
(Le., subjective or in-complete) queries, through the so-called "fuzzification" technique, which 
translates the feature space to a fuzzy space. Also see page 440 the last paragraph before 
section 4.2 Euripides discloses set of alphabet graph or image translated or scaled with respect 
to each other. 

Claims 7 and 8. 

The computerized method of claim 4, wherein the applying comprises: performing 
pushout/pullback operations. As Applicant on page 11 paragraph 0029 discloses that the two 
particular operations in algebraic graph grammars are suitable to build graph rewriting 
techniques for description scheme graphs: pushout and pullback. Pushouts and pullbacks can be 
thought of as sums and products, respectively. Euripides on page 437 in Eqs. 1 -4 clearly shows 
the meaning of sums and products. Also in fig. lb illustrates the pullback (also called the fiber 

product) is the limit of a diagram consisting of at least two morphisms r23: v2 -+ v3 and rl2: 

vl v3 with a common codomain (A set within which the values of a function lie (as opposed 

to the range, which is the set of values that the function actually takes). Explicitly, the 
pullback of the morphisms r23 and rl2 consists of an object vO and two morphisms rOl: vO -+ 

vl and r02: vO -> v3 for which the diagram. The categorical dual of a pullback is a called a 
pushout. 

Claim 9. 
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The computerized method of claim 8, wherein performing the pullback operation comprises: 
creating adjacency matrices representing smallest portions of the set of morphism graphs that 
map the input and rule graphs to the alphabet graph using pre-images of parts of the alphabet 
graph marked for change; and multiplying the adjacency matrix associated with the input graph 
by a transpose of the adjacency matrix associated with the rule graph. See rejection of claims 7-8 
that applies to the rejection of this claim. 

Claim 10 

See rejection of claim 1 . 

Claims 11-18 

The rejections of claims 2-9 apply to the rejections of claims 11-18. 
Claims 19-27 

The rejections of claims 2-9 apply to the rejections of claims 19-27. 
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Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Javid A. Amini whose telephone number is 571-272-7654. The 
examiner can normally be reached on 8-4pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi can be reached on 571-272-7664. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR, 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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